Add validation + compatibility tests for DeepLabCut-live#56
Open
deruyter92 wants to merge 16 commits intocy/pre-release-fixes-2.0from
Open
Add validation + compatibility tests for DeepLabCut-live#56deruyter92 wants to merge 16 commits intocy/pre-release-fixes-2.0from
deruyter92 wants to merge 16 commits intocy/pre-release-fixes-2.0from
Conversation
deruyter92
changed the title
deruyter92
changed the title
C-Achard
reviewed
Feb 27, 2026
There was a problem hiding this comment.
@deruyter92 If you'd like I can merge #55 right now, so then we can see how the tests go in CI ?
Author
|
Yes great thanks. Sorry I see that I didn't install the pre-commit |
|
Sorry, in the above I meant #55. It is now merged, happy to take care of conflicts/CI if you're busy ! |
20 tasks
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>
Co-authored-by: Cyril Achard <cyril.achard@epfl.ch>
C-Achard
force-pushed
the
jaap/add_compatibility_tests
branch
from
54ee55e to
693c931
Compare
Remove the earlier pytest commands block and update the tox 'commands' to run pytest with the marker excluding both 'hardware' and 'dlclive_compat'. Adjust coverage invocation to use the installed package path (--cov={envsitepackagesdir}/dlclivegui) and emit per-env XML coverage files (.coverage.{envname}.xml). This aligns tox behavior with the GitHub Actions job and removes the prior posargs-based command duplication.
C-Achard
reviewed
Feb 27, 2026
Update the GitHub Actions testing matrix to run on Python 3.12 instead of 3.11. This moves CI to test against the newer Python runtime while keeping existing matrix include entries unchanged.
Author
|
I see that some tests are failing still. I am happy to pick this up later. Importantly this shouldn't hold you back from releasing the bèta-test version. These validation measures can always be added later. |
Add a PoseBackends Enum and switch PoseSource.backend from a string to that enum for stronger typing and clarity. Update default PosePacket/PoseSource instances and the validate_pose_array signature to use PoseBackends.DLC_LIVE. Import Enum/auto and remove an unused sentinel variable. These are small refactors to improve type-safety and consistency around backend identification.
Relax and correct DLCLive compatibility checks: only consider keyword/positional params when inspecting __init__, drop the assertion that __init__ must accept **kwargs, and only require 'frame' for init_inference/get_pose (frame_time is passed as a kwarg to processors). Also update FakeDLCLive.get_pose to return an array of shape (2, 3) to match the expected pose output shape.
Set the test step shell to `bash -eo pipefail` and redirect `tox` stderr into `tee` (changed `tox -q` to `tox -q 2>&1 | tee tox-output.log`). This ensures pipeline failures are detected (pipefail) and that tox's stderr is recorded in `tox-output.log` for improved debugging in the CI workflow.
Replace ad-hoc DLCLive installs in the GitHub Actions job with tox-based testenvs. The workflow fixes Python to 3.12, installs required Qt/OpenGL runtime libs on Ubuntu, installs tox and tox-gh-actions, and runs tox -e matrix.tox_env. tox.ini was extended with dlclive-pypi and dlclive-github testenvs (pypi pinned and GitHub main respectively) to run the compatibility pytest, and the new envs were added to env_list to allow local and CI execution.
|
@deruyter92 A few additions :
|
Add tests/compat/conftest.py to inject a stub torch module into sys.modules when torch is not installed. This prevents ImportError during DLCLive compatibility tests so the API can be validated without requiring torch to be installed. This is a pragmatic workaround and includes a note to remove or replace it once imports are properly guarded in the package.
|
I also had to monkeypatch torch in the compat tests due to unguarded imports in DLCLive, see DeepLabCut/DeepLabCut-live#168 |
Replace QTimer.singleShot with a cancellable QTimer instance (self._camera_validation_timer) that is single-shot, connected to _validate_configured_cameras, and started with a 100ms delay. The closeEvent now stops the timer if it's still active to prevent validation from firing while the window is closing, avoiding modal QMessageBox races/crashes during tests/CI teardown.
C-Achard
added
DLClive
C-Achard
approved these changes
Mar 2, 2026
There was a problem hiding this comment.
Approving again, will wait for your feedback next week @deruyter92 before merging as this is agreed not to be beta-critical; thanks again for all the valuable additions !
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation:
Since DeepLabCut-live-GUI heavily depends on DeepLabCut-live, but currenly it is implicitly assumed that that package provides the right output. It would be good to add some basic explicit validation of the returned output of DeepLabCut-live to check compatibility of the returned data. Also, including functional compatibility checks in CI for different DeepLabcut-live versions, allows to systematically tets the package compatibility before release.
Changes:
PosePacketa lightweight schema validation of the output from DeepLabCut-liveThis pull request introduces significant improvements to DLCLive compatibility testing and pose validation in both the codebase and CI infrastructure. The main changes include adding a dedicated compatibility test suite for DLCLive versions, implementing robust pose array validation, and updating the workflow and test environments to support these enhancements.
DLCLive compatibility testing:
tests/compat/test_dlclive_package_compat.pythat verifies DLCLive package importability, constructor signatures, required methods, and performs a minimal inference smoke test. This ensures that the GUI remains compatible with supported DLCLive versions.torchintests/compat/conftest.pyto allow compatibility tests to run even iftorchis not installed, improving test robustness.Pose validation and contract enforcement:
validate_pose_arrayfunction indlclivegui/services/dlc_processor.pyto enforce shape, dtype, and value constraints on pose outputs, preventing downstream errors from invalid data.tests/services/test_pose_contract.py, covering both valid and invalid cases.Continuous integration and test environment updates:
dlclive-compatjob to.github/workflows/testing-ci.ymlto run compatibility tests against multiple DLCLive versions (PyPI and GitHub main), and updated tox configuration to include corresponding environments. [1] [2] [3]pyproject.tomlto includedlclive_compatand excluded these tests from the default coverage run. [1] [2]GUI camera validation timer improvements:
dlclivegui/gui/main_window.pyto use a cancellable QTimer attribute and ensure it is stopped incloseEvent, reducing test/CI errors related to GUI teardown timing. [1] [2]These changes improve reliability, maintainability, and future-proofing of the DLCLive integration and pose processing pipeline.